Regularization and the small-ball method II: complexity dependent error rates

For a convex class of functions F , a regularization functions Ψ(·) and given the random data (Xi, Yi) N i=1, we study estimation properties of regularization procedures of the form f̂ ∈ argmin f∈F ( 1 N N ∑ i=1 ( Yi − f(Xi) )2 + λΨ(f) ) for some well chosen regularization parameter λ. We obtain bounds on the L2 estimation error rate that depend on the complexity of the “true model” F ∗ := {f ∈ F : Ψ(f) ≤ Ψ(f∗)}, where f∗ ∈ argminf∈F E(Y −f(X)) and the (Xi, Yi)’s are independent and distributed as (X,Y ). Our estimate holds under weak stochastic assumptions – one of which being a small-ball condition satisfied by F – and for rather flexible choices of regularization functions Ψ(·). Moreover, the result holds in the learning theory framework: we do not assume any a-priori connection between the output Y and the input X). As a proof of concept, we apply our general estimation bound to various choices of Ψ, for example, the `p and Sp-norms (for p ≥ 1), weak-`p, atomic norms, max-norm and SLOPE. In many cases, the estimation rate almost coincides with the minimax rate in the class F ∗.